Machine Learning review and intro to tidymodels
Read through and follow along with the Machine Learning review with an intro to the tidymodels package posted on the Course Materials page.
Tasks:
- Read about the hotel booking data,
hotels, on the Tidy Tuesday page it came from. There is also a link to an article from the original authors. The outcome we will be predicting is called is_canceled.
- Without doing any analysis, what are some variables you think might be predictive and why?
lead_time. Just intuitively, the closer up to the arrival date the less likely the appointment might get cancelled. previous_cancellations. The guest might more likely to cancel the appointment if they have already done it before. previous_bookings_not_canceled. If previous booking wasn't cancelled, it is probably more likely that this one won't be cancelled as well. deposit_type. If the deposit wasn't refundable, the guest is less likely to cancel the bookings.
What are some problems that might exist with the data? You might think about how it was collected and who did the collecting.
The dataset might not be randomly collected, and the collector might already have pre-supposed assumptions about the dataset. Moreover, the sample in the dataset is a bit inbalanced.
If we construct a model, what type of conclusions will be able to draw from it?
We will be able to know what type of hotel bookings is more likely to be cancelled.
- Create some exploratory plots or table summaries of the variables in the dataset. Be sure to also examine missing values or other interesting values. You may want to adjust the
fig.width and fig.height in the code chunk options.
hotels %>%
select(where(is.numeric)) %>%
pivot_longer(cols = everything(),
names_to = "variable",
values_to = "value") %>%
ggplot(aes(x = value)) +
geom_histogram(bins = 30) +
facet_wrap(vars(variable),
scales = "free")

lead_time is very right-skewed, same as n_weekend. Nearly all hotel has very small sumber of repeated guests.
hotels %>%
#select(where(is.factor)) %>%
select(where(is.character)) %>%
pivot_longer(cols = everything(),
names_to = "variable",
values_to = "value") %>%
ggplot(aes(x = value)) +
geom_bar() +
facet_wrap(vars(variable),
scales = "free",
nrow = 2)

Almost all dates of month have a significant amount of bookings, while the first few dates have more number of bookings at the biginning of month.
- First, we will do a couple things to get the data ready.
I did the following for you: made outcome a factor (needs to be that way for logistic regression), made all character variables factoors, removed the year variable and some reservation status variables, and removed cases with missing values (not NULLs but true missing values).
You need to split the data into a training and test set, stratifying on the outcome variable, is_canceled. Since we have a lot of data, split the data 50/50 between training and test. I have already set.seed() for you. Be sure to use hotels_mod in the splitting.
hotels_mod <- hotels %>%
mutate(is_canceled = as.factor(is_canceled)) %>%
mutate(across(where(is.character), as.factor)) %>%
select(-arrival_date_year,
-reservation_status,
-reservation_status_date) %>%
add_n_miss() %>%
filter(n_miss_all == 0) %>%
select(-n_miss_all)
set.seed(494)
hotel_split <- initial_split(hotels_mod, prop = .50)
hotel_training <- training(hotel_split)
hotel_testing <- testing(hotel_split)
- In this next step, we are going to do the pre-processing. Usually, I won't tell you exactly what to do here, but for your first exercise, I'll tell you the steps.
- Set up the recipe with
is_canceled as the outcome and all other variables as predictors (HINT: ~.).
- Use a
step_XXX() function or functions (I think there are other ways to do this, but I found step_mutate_at() easiest) to create some indicator variables for the following variables: children, babies, and previous_cancellations. So, the new variable should be a 1 if the original is more than 0 and 0 otherwise. Make sure you do this in a way that accounts for values that may be larger than any we see in the dataset.
- For the
agent and company variables, make new indicator variables that are 1 if they have a value of NULL and 0 otherwise. I also used step_mutate_at() for this, but there's more ways you could do it.
- Use
fct_lump_n() inside step_mutate() to lump together countries that aren't in the top 5 most occurring.
- If you used new names for some of the new variables you created, then remove any variables that are no longer needed.
- Use
step_normalize() to center and scale all the non-categorical predictor variables. (Do this BEFORE creating dummy variables. When I tried to do it after, I ran into an error - I'm still investigating why.)
- Create dummy variables for all factors/categorical predictor variables (make sure you have
-all_outcomes() in this part!!).
- Use the
prep() and juice() functions to apply the steps to the training data just to check that everything went as planned.
# hotel_recipe <- recipe(is_canceled ~ .,
# data = hotel_training) %>%
# step_mutate_at(children, babies, previous_cancellations, fn = ~ ifelse(.>0, 1, 0)) %>%
# step_normalize(all_numeric_predictors())%>%
# step_dummy(all_nominal(),-all_outcomes())
#
# hotel_recipe %>%
# prep(hotel_training) %>%
# juice()
hotel_recipe<-recipe(is_canceled~.,
data = hotel_training) %>%
step_mutate(children = ifelse(children > 0, 1, 0),
babies = ifelse(babies > 0, 1, 0),
previous_cancellations = ifelse(previous_cancellations >0, 1, 0),
agent = ifelse(agent == "NULL", 1, 0 ),
company = ifelse(company == "NULL", 1, 0),
country = fct_lump_n(country, n=5) ) %>%
step_normalize(all_numeric_predictors())%>%#all_predictors(),
#-all_nominal(),
#-all_outcomes() %>%
step_dummy(all_nominal(),
-all_outcomes())#all_nominal_predictors
hotel_recipe %>%
prep(hotel_training) %>%
juice()
- In this step we will set up a LASSO model and workflow.
- In general, why would we want to use LASSO instead of regular logistic regression? (HINT: think about what happens to the coefficients).
Using regular logistic regression models is particularly problematic when there are a large number of features, which can make the model less interpretable. Using LASSO, we are able to incorporate feature selection, since it can push the effect of irrelevant features toward zero. Thus, it can not only improve the model but it also conduct automated feature selection.
- Define the model type, set the engine, set the
penalty argument to tune() as a placeholder, and set the mode.
hotel_lasso_mod <-
logistic_reg(mixture = 1) %>%
set_engine("glmnet") %>%
set_args(penalty = tune()) %>%
set_mode("classification")
- Create a workflow with the recipe and model.
hotel_lasso_wf <-
workflow() %>%
add_recipe(hotel_recipe) %>%
add_model(hotel_lasso_mod)
- In this step, we'll tune the model and fit the model using the best tuning parameter to the entire training dataset.
- Create a 5-fold cross-validation sample. We'll use this later. I have set the seed for you.
- Use the
grid_regular() function to create a grid of 10 potential penalty parameters (we're keeping this sort of small because the dataset is pretty large). Use that with the 5-fold cv data to tune the model.
- Use the
tune_grid() function to fit the models with different tuning parameters to the different cross-validation sets.
- Use the
collect_metrics() function to collect all the metrics from the previous step and create a plot with the accuracy on the y-axis and the penalty term on the x-axis. Put the x-axis on the log scale.
- Use the
select_best() function to find the best tuning parameter, fit the model using that tuning parameter to the entire training set (HINT: finalize_workflow() and fit()), and display the model results using pull_workflow_fit() and tidy(). Are there some variables with coefficients of 0?
set.seed(494) # for reproducibility
hotel_cv <- vfold_cv(hotel_training, v = 5)
penalty_grid <- grid_regular(penalty(),
levels = 10)
ctrl_grid <- control_stack_grid()
#metric <- metric_set(rmse)
hotel_lasso_tune <-
hotel_lasso_wf %>%
tune_grid(
resamples = hotel_cv,
grid = penalty_grid,
#metrics = metric,
control = ctrl_grid
)
hotel_lasso_tune %>%
collect_metrics() %>%
filter(.metric == "accuracy") %>%
ggplot(aes(x = penalty, y = mean)) +
geom_point() +
geom_line() +
scale_x_log10(
breaks = scales::trans_breaks("log10", function(x) 10^x),
labels = scales::trans_format("log10",scales::math_format(10^.x))) +
labs(x = "penalty", y = "accuracy")

best_param <- hotel_lasso_tune %>%
select_best(metric = "accuracy")
best_param
one_se_param <- hotel_lasso_tune %>%
select_by_one_std_err(metric = "accuracy", desc(penalty))
hotel_lasso_final_wf <- hotel_lasso_wf %>%
finalize_workflow(one_se_param)
hotel_lasso_final_wf
## ══ Workflow ════════════════════════════════════════════════════════════════════
## Preprocessor: Recipe
## Model: logistic_reg()
##
## ── Preprocessor ────────────────────────────────────────────────────────────────
## 3 Recipe Steps
##
## • step_mutate()
## • step_normalize()
## • step_dummy()
##
## ── Model ───────────────────────────────────────────────────────────────────────
## Logistic Regression Model Specification (classification)
##
## Main Arguments:
## penalty = 0.000464158883361278
## mixture = 1
##
## Computational engine: glmnet
hotel_lasso_final_mod <- hotel_lasso_final_wf %>%
fit(data = hotel_training)
hotel_lasso_final_mod %>%
pull_workflow_fit() %>%
tidy()
- Now that we have a model, let's evaluate it a bit more. All we have looked at so far is the cross-validated accuracy from the previous step.
- Create a variable importance graph. Which variables show up as the most important? Are you surprised?
hotel_lasso_final_mod %>%
pull_workflow_fit() %>%
vip()

The most important variable is reserved room type. Yes I am surprised.
- Use the
last_fit() function to fit the final model and then apply it to the testing data. Report the metrics from the testing data using the collet_metrics() function. How do they compare to the cross-validated metrics?
# Fit model with best tuning parameter(s) to training data and apply to test data
hotel_lasso_test <- hotel_lasso_final_wf %>%
last_fit(hotel_split)
# Metrics for model applied to test data
hotel_lasso_test %>%
collect_metrics()
collect_metrics(hotel_lasso_tune)
The accuracy of testing data slight decreases comparing to cross-validated metrics.
- Use the
collect_predictions() function to find the predicted probabilities and classes for the test data. Save this to a new dataset called preds. Then, use the conf_mat() function from dials (part of tidymodels) to create a confusion matrix showing the predicted classes vs. the true classes. Compute the true positive rate (sensitivity), true negative rate (specificity), and accuracy. See this Wikipedia reference if you (like me) tend to forget these definitions. Also keep in mind that a "positive" in this case is a cancellation (those are the 1's).
preds <- collect_predictions(hotel_lasso_test)
preds %>%
conf_mat(is_canceled, .pred_class)
## Truth
## Prediction 0 1
## 0 34304 7806
## 1 3297 14286
true positive rate (sensitivity):
14286/(14286 + 3297)
## [1] 0.8124893
true negative rate (specificity):
34304/(34304+7806)
## [1] 0.8146284
- Use the
preds dataset you just created to create a density plot of the predicted probabilities of canceling (the variable is called .pred_1), filling by is_canceled. Use an alpha = .5 and color = NA in the geom_density().
preds %>%
ggplot(aes(x=.pred_1)) +
geom_density(aes(fill = is_canceled), alpha = .5, color = NA)

Answer these questions: a. What would this graph look like for a model with an accuracy that was close to 1?
The red part (is_cancelled 0) would be all close to 0, meaning .pred_1 = 0 , while the blue part would be close to 1, meaning .pred_1 = 1.
- Our predictions are classified as canceled if their predicted probability of canceling is greater than .5. If we wanted to have a high true positive rate, should we make the cutoff for predicted as canceled higher or lower than .5?
Lower. Since the true positive rate is the probability that an actual positive will test positive, if the cutoff is lower, more cases will be classfied as positive and thus the true positive rate will be higher.
- What happens to the true negative rate if we try to get a higher true positive rate?
The true negative rate will decrease when we try to get a higher true positive rate. They are negatively correlated.
- Let's say that this model is going to be applied to bookings 14 days in advance of their arrival at each hotel, and someone who works for the hotel will make a phone call to the person who made the booking. During this phone call, they will try to assure that the person will be keeping their reservation or that they will be canceling in which case they can do that now and still have time to fill the room. How should the hotel go about deciding who to call? How could they measure whether it was worth the effort to do the calling? Can you think of another way they might use the model?
The hotel can use the model result to look at who to call, if the model predicts the customer as cancelled, then it probably should make a phone call to this person.
We could look at the results from collect_predictions and compare the numbers of .pred_0 and .pred_1. If we are getting a 1 in .pred_class, we should look at .pred_1 to see the chance of getting 1, if it is high, then we know that this person is worth the effort to do the calling since the chance of getting a 1 is high.
Another way to use this model is to look at feature importance, and without accually implementing the model on new data, we might be able to infer something from important features.
- How might you go about questioning and evaluating the model in terms of fairness? Are there any questions you would like to ask of the people who collected the data?
Is the distribution of race, age and gender even in the dataset? Is there any discrimination of age, age and gender in the data collection process? Is the data collected ethically and properly with consent from concerning parties?
LS0tCnRpdGxlOiAnQXNzaWdubWVudCAjMicKb3V0cHV0OiAKICBodG1sX2RvY3VtZW50OgogICAgdG9jOiB0cnVlCiAgICB0b2NfZmxvYXQ6IHRydWUKICAgIGRmX3ByaW50OiBwYWdlZAogICAgY29kZV9kb3dubG9hZDogdHJ1ZQotLS0KCmBgYHtyIHNldHVwLCBpbmNsdWRlPUZBTFNFfQprbml0cjo6b3B0c19jaHVuayRzZXQoZWNobyA9IFRSVUUsIG1lc3NhZ2U9RkFMU0UsIHdhcm5pbmc9RkFMU0UpCmBgYAoKYGBge3IgbGlicmFyaWVzfQpsaWJyYXJ5KHRpZHl2ZXJzZSkgICAgICAgICAjIGZvciBncmFwaGluZyBhbmQgZGF0YSBjbGVhbmluZwpsaWJyYXJ5KHRpZHltb2RlbHMpICAgICAgICAjIGZvciBtb2RlbGluZwpsaWJyYXJ5KG5hbmlhcikgICAgICAgICAgICAjIGZvciBhbmFseXppbmcgbWlzc2luZyB2YWx1ZXMKbGlicmFyeSh2aXApICAgICAgICAgICAgICAgIyBmb3IgdmFyaWFibGUgaW1wb3J0YW5jZSBwbG90cwp0aGVtZV9zZXQodGhlbWVfbWluaW1hbCgpKSAjIExpc2EncyBmYXZvcml0ZSB0aGVtZQpsaWJyYXJ5KHN0YWNrcykgIApgYGAKCmBgYHtyIGRhdGF9CmhvdGVscyA8LSByZWFkcjo6cmVhZF9jc3YoJ2h0dHBzOi8vcmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbS9yZm9yZGF0YXNjaWVuY2UvdGlkeXR1ZXNkYXkvbWFzdGVyL2RhdGEvMjAyMC8yMDIwLTAyLTExL2hvdGVscy5jc3YnKQpgYGAKCgpXaGVuIHlvdSBmaW5pc2ggdGhlIGFzc2lnbm1lbnQsIHJlbW92ZSB0aGUgYCNgIGZyb20gdGhlIG9wdGlvbnMgY2h1bmsgYXQgdGhlIHRvcCwgc28gdGhhdCBtZXNzYWdlcyBhbmQgd2FybmluZ3MgYXJlbid0IHByaW50ZWQuIElmIHlvdSBhcmUgZ2V0dGluZyBlcnJvcnMgaW4geW91ciBjb2RlLCBhZGQgYGVycm9yID0gVFJVRWAgc28gdGhhdCB0aGUgZmlsZSBrbml0cy4gSSB3b3VsZCByZWNvbW1lbmQgbm90IHJlbW92aW5nIHRoZSBgI2AgdW50aWwgeW91IGFyZSBjb21wbGV0ZWx5IGZpbmlzaGVkLgoKIyMgUHV0IGl0IG9uIEdpdEh1YiEgICAgICAgIAoKRnJvbSBub3cgb24sIEdpdEh1YiBzaG91bGQgYmUgcGFydCBvZiB5b3VyIHJvdXRpbmUgd2hlbiBkb2luZyBhc3NpZ25tZW50cy4gSSByZWNvbW1lbmQgbWFraW5nIGl0IHBhcnQgb2YgeW91ciBwcm9jZXNzIGFueXRpbWUgeW91IGFyZSB3b3JraW5nIGluIFIsIGJ1dCBJJ2xsIG1ha2UgeW91IHNob3cgaXQncyBwYXJ0IG9mIHlvdXIgcHJvY2VzcyBmb3IgYXNzaWdubWVudHMuCgoqKlRhc2sqKjogV2hlbiB5b3UgYXJlIGZpbmlzaGVkIHdpdGggdGhlIGFzc2lnbm1lbnQsIHBvc3QgYSBsaW5rIGJlbG93IHRvIHRoZSBHaXRIdWIgcmVwbyBmb3IgdGhlIGFzc2lnbm1lbnQuIAoKW0hvbWV3b3JrIDJdKGh0dHBzOi8vZ2l0aHViLmNvbS9raWRha2svQWR2YW5jZWQtRGF0YS1TY2llbmNlL2Jsb2IvbWFpbi8wMl9hc3NpZ25tZW50LlJtZCkgCgojIyBNYWNoaW5lIExlYXJuaW5nIHJldmlldyBhbmQgaW50cm8gdG8gYHRpZHltb2RlbHNgCgpSZWFkIHRocm91Z2ggYW5kIGZvbGxvdyBhbG9uZyB3aXRoIHRoZSBbTWFjaGluZSBMZWFybmluZyByZXZpZXcgd2l0aCBhbiBpbnRybyB0byB0aGUgYHRpZHltb2RlbHNgIHBhY2thZ2VdKGh0dHBzOi8vYWR2YW5jZWQtZHMtaW4tci5uZXRsaWZ5LmFwcC9wb3N0cy8yMDIxLTAzLTE2LW1sLXJldmlldy8pIHBvc3RlZCBvbiB0aGUgQ291cnNlIE1hdGVyaWFscyBwYWdlLiAKCioqVGFza3MqKjoKCjEuIFJlYWQgYWJvdXQgdGhlIGhvdGVsIGJvb2tpbmcgZGF0YSwgYGhvdGVsc2AsIG9uIHRoZSBbVGlkeSBUdWVzZGF5IHBhZ2VdKGh0dHBzOi8vZ2l0aHViLmNvbS9yZm9yZGF0YXNjaWVuY2UvdGlkeXR1ZXNkYXkvYmxvYi9tYXN0ZXIvZGF0YS8yMDIwLzIwMjAtMDItMTEvcmVhZG1lLm1kKSBpdCBjYW1lIGZyb20uIFRoZXJlIGlzIGFsc28gYSBsaW5rIHRvIGFuIGFydGljbGUgZnJvbSB0aGUgb3JpZ2luYWwgYXV0aG9ycy4gVGhlIG91dGNvbWUgd2Ugd2lsbCBiZSBwcmVkaWN0aW5nIGlzIGNhbGxlZCBgaXNfY2FuY2VsZWRgLiAKICAtIFdpdGhvdXQgZG9pbmcgYW55IGFuYWx5c2lzLCB3aGF0IGFyZSBzb21lIHZhcmlhYmxlcyB5b3UgdGhpbmsgbWlnaHQgYmUgcHJlZGljdGl2ZSBhbmQgd2h5PyAKICAKICBsZWFkX3RpbWUuIEp1c3QgaW50dWl0aXZlbHksIHRoZSBjbG9zZXIgdXAgdG8gdGhlIGFycml2YWwgZGF0ZSB0aGUgbGVzcyBsaWtlbHkgdGhlIGFwcG9pbnRtZW50IG1pZ2h0IGdldCBjYW5jZWxsZWQuIAogIHByZXZpb3VzX2NhbmNlbGxhdGlvbnMuIFRoZSBndWVzdCBtaWdodCBtb3JlIGxpa2VseSB0byBjYW5jZWwgdGhlIGFwcG9pbnRtZW50IGlmIHRoZXkgaGF2ZSBhbHJlYWR5IGRvbmUgaXQgYmVmb3JlLiAKICBwcmV2aW91c19ib29raW5nc19ub3RfY2FuY2VsZWQuIElmIHByZXZpb3VzIGJvb2tpbmcgd2Fzbid0IGNhbmNlbGxlZCwgaXQgaXMgcHJvYmFibHkgbW9yZSBsaWtlbHkgdGhhdCB0aGlzIG9uZSB3b24ndCBiZSBjYW5jZWxsZWQgYXMgd2VsbC4gCiAgZGVwb3NpdF90eXBlLiBJZiB0aGUgZGVwb3NpdCB3YXNuJ3QgcmVmdW5kYWJsZSwgdGhlIGd1ZXN0IGlzIGxlc3MgbGlrZWx5IHRvIGNhbmNlbCB0aGUgYm9va2luZ3MuIAoKICAtIFdoYXQgYXJlIHNvbWUgcHJvYmxlbXMgdGhhdCBtaWdodCBleGlzdCB3aXRoIHRoZSBkYXRhPyBZb3UgbWlnaHQgdGhpbmsgYWJvdXQgaG93IGl0IHdhcyBjb2xsZWN0ZWQgYW5kIHdobyBkaWQgdGhlIGNvbGxlY3RpbmcuICAKICBUaGUgZGF0YXNldCBtaWdodCBub3QgYmUgcmFuZG9tbHkgY29sbGVjdGVkLCBhbmQgdGhlIGNvbGxlY3RvciBtaWdodCBhbHJlYWR5IGhhdmUgcHJlLXN1cHBvc2VkIGFzc3VtcHRpb25zIGFib3V0IHRoZSBkYXRhc2V0LiBNb3Jlb3ZlciwgdGhlIHNhbXBsZSBpbiB0aGUgZGF0YXNldCBpcyBhIGJpdCBpbmJhbGFuY2VkLiAKCiAgLSBJZiB3ZSBjb25zdHJ1Y3QgYSBtb2RlbCwgd2hhdCB0eXBlIG9mIGNvbmNsdXNpb25zIHdpbGwgYmUgYWJsZSB0byBkcmF3IGZyb20gaXQ/ICAKICBXZSB3aWxsIGJlIGFibGUgdG8ga25vdyB3aGF0IHR5cGUgb2YgaG90ZWwgYm9va2luZ3MgaXMgbW9yZSBsaWtlbHkgdG8gYmUgY2FuY2VsbGVkLiAKICAKMi4gQ3JlYXRlIHNvbWUgZXhwbG9yYXRvcnkgcGxvdHMgb3IgdGFibGUgc3VtbWFyaWVzIG9mIHRoZSB2YXJpYWJsZXMgaW4gdGhlIGRhdGFzZXQuIEJlIHN1cmUgdG8gYWxzbyBleGFtaW5lIG1pc3NpbmcgdmFsdWVzIG9yIG90aGVyIGludGVyZXN0aW5nIHZhbHVlcy4gWW91IG1heSB3YW50IHRvIGFkanVzdCB0aGUgYGZpZy53aWR0aGAgYW5kIGBmaWcuaGVpZ2h0YCBpbiB0aGUgY29kZSBjaHVuayBvcHRpb25zLiAgCgpgYGB7cn0KaG90ZWxzICU+JSAKICBzZWxlY3Qod2hlcmUoaXMubnVtZXJpYykpICU+JSAKICBwaXZvdF9sb25nZXIoY29scyA9IGV2ZXJ5dGhpbmcoKSwKICAgICAgICAgICAgICAgbmFtZXNfdG8gPSAidmFyaWFibGUiLCAKICAgICAgICAgICAgICAgdmFsdWVzX3RvID0gInZhbHVlIikgJT4lIAogIGdncGxvdChhZXMoeCA9IHZhbHVlKSkgKwogIGdlb21faGlzdG9ncmFtKGJpbnMgPSAzMCkgKwogIGZhY2V0X3dyYXAodmFycyh2YXJpYWJsZSksIAogICAgICAgICAgICAgc2NhbGVzID0gImZyZWUiKQpgYGAKCmxlYWRfdGltZSBpcyB2ZXJ5IHJpZ2h0LXNrZXdlZCwgc2FtZSBhcyBuX3dlZWtlbmQuIE5lYXJseSBhbGwgaG90ZWwgaGFzIHZlcnkgc21hbGwgc3VtYmVyIG9mIHJlcGVhdGVkIGd1ZXN0cy4gCgpgYGB7cn0KaG90ZWxzICU+JSAKICAjc2VsZWN0KHdoZXJlKGlzLmZhY3RvcikpICU+JSAKICBzZWxlY3Qod2hlcmUoaXMuY2hhcmFjdGVyKSkgJT4lIAogIHBpdm90X2xvbmdlcihjb2xzID0gZXZlcnl0aGluZygpLAogICAgICAgICAgICAgICBuYW1lc190byA9ICJ2YXJpYWJsZSIsIAogICAgICAgICAgICAgICB2YWx1ZXNfdG8gPSAidmFsdWUiKSAlPiUgCiAgZ2dwbG90KGFlcyh4ID0gdmFsdWUpKSArCiAgZ2VvbV9iYXIoKSArCiAgZmFjZXRfd3JhcCh2YXJzKHZhcmlhYmxlKSwgCiAgICAgICAgICAgICBzY2FsZXMgPSAiZnJlZSIsIAogICAgICAgICAgICAgbnJvdyA9IDIpCmBgYAoKQWxtb3N0IGFsbCBkYXRlcyBvZiBtb250aCBoYXZlIGEgc2lnbmlmaWNhbnQgYW1vdW50IG9mIGJvb2tpbmdzLCB3aGlsZSB0aGUgZmlyc3QgZmV3IGRhdGVzIGhhdmUgbW9yZSBudW1iZXIgb2YgYm9va2luZ3MgYXQgdGhlIGJpZ2lubmluZyBvZiBtb250aC4gCgozLiBGaXJzdCwgd2Ugd2lsbCBkbyBhIGNvdXBsZSB0aGluZ3MgdG8gZ2V0IHRoZSBkYXRhIHJlYWR5LiAKCiogSSBkaWQgdGhlIGZvbGxvd2luZyBmb3IgeW91OiBtYWRlIG91dGNvbWUgYSBmYWN0b3IgKG5lZWRzIHRvIGJlIHRoYXQgd2F5IGZvciBsb2dpc3RpYyByZWdyZXNzaW9uKSwgbWFkZSBhbGwgY2hhcmFjdGVyIHZhcmlhYmxlcyBmYWN0b29ycywgcmVtb3ZlZCB0aGUgeWVhciB2YXJpYWJsZSBhbmQgc29tZSByZXNlcnZhdGlvbiBzdGF0dXMgdmFyaWFibGVzLCBhbmQgcmVtb3ZlZCBjYXNlcyB3aXRoIG1pc3NpbmcgdmFsdWVzIChub3QgTlVMTHMgYnV0IHRydWUgbWlzc2luZyB2YWx1ZXMpLgoKKiBZb3UgbmVlZCB0byBzcGxpdCB0aGUgZGF0YSBpbnRvIGEgdHJhaW5pbmcgYW5kIHRlc3Qgc2V0LCBzdHJhdGlmeWluZyBvbiB0aGUgb3V0Y29tZSB2YXJpYWJsZSwgYGlzX2NhbmNlbGVkYC4gU2luY2Ugd2UgaGF2ZSBhIGxvdCBvZiBkYXRhLCBzcGxpdCB0aGUgZGF0YSA1MC81MCBiZXR3ZWVuIHRyYWluaW5nIGFuZCB0ZXN0LiBJIGhhdmUgYWxyZWFkeSBgc2V0LnNlZWQoKWAgZm9yIHlvdS4gQmUgc3VyZSB0byB1c2UgYGhvdGVsc19tb2RgIGluIHRoZSBzcGxpdHRpbmcuCgpgYGB7cn0KaG90ZWxzX21vZCA8LSBob3RlbHMgJT4lIAogIG11dGF0ZShpc19jYW5jZWxlZCA9IGFzLmZhY3Rvcihpc19jYW5jZWxlZCkpICU+JSAKICBtdXRhdGUoYWNyb3NzKHdoZXJlKGlzLmNoYXJhY3RlciksIGFzLmZhY3RvcikpICU+JSAKICBzZWxlY3QoLWFycml2YWxfZGF0ZV95ZWFyLAogICAgICAgICAtcmVzZXJ2YXRpb25fc3RhdHVzLAogICAgICAgICAtcmVzZXJ2YXRpb25fc3RhdHVzX2RhdGUpICU+JSAKICBhZGRfbl9taXNzKCkgJT4lIAogIGZpbHRlcihuX21pc3NfYWxsID09IDApICU+JSAKICBzZWxlY3QoLW5fbWlzc19hbGwpCgpzZXQuc2VlZCg0OTQpCgpob3RlbF9zcGxpdCA8LSBpbml0aWFsX3NwbGl0KGhvdGVsc19tb2QsIHByb3AgPSAuNTApCgpob3RlbF90cmFpbmluZyA8LSB0cmFpbmluZyhob3RlbF9zcGxpdCkKaG90ZWxfdGVzdGluZyA8LSB0ZXN0aW5nKGhvdGVsX3NwbGl0KQpgYGAKCjQuIEluIHRoaXMgbmV4dCBzdGVwLCB3ZSBhcmUgZ29pbmcgdG8gZG8gdGhlIHByZS1wcm9jZXNzaW5nLiBVc3VhbGx5LCBJIHdvbid0IHRlbGwgeW91IGV4YWN0bHkgd2hhdCB0byBkbyBoZXJlLCBidXQgZm9yIHlvdXIgZmlyc3QgZXhlcmNpc2UsIEknbGwgdGVsbCB5b3UgdGhlIHN0ZXBzLiAKCiogU2V0IHVwIHRoZSByZWNpcGUgd2l0aCBgaXNfY2FuY2VsZWRgIGFzIHRoZSBvdXRjb21lIGFuZCBhbGwgb3RoZXIgdmFyaWFibGVzIGFzIHByZWRpY3RvcnMgKEhJTlQ6IGB+LmApLiAgCiogVXNlIGEgYHN0ZXBfWFhYKClgIGZ1bmN0aW9uIG9yIGZ1bmN0aW9ucyAoSSB0aGluayB0aGVyZSBhcmUgb3RoZXIgd2F5cyB0byBkbyB0aGlzLCBidXQgSSBmb3VuZCBgc3RlcF9tdXRhdGVfYXQoKWAgZWFzaWVzdCkgdG8gY3JlYXRlIHNvbWUgaW5kaWNhdG9yIHZhcmlhYmxlcyBmb3IgdGhlIGZvbGxvd2luZyB2YXJpYWJsZXM6IGBjaGlsZHJlbmAsIGBiYWJpZXNgLCBhbmQgYHByZXZpb3VzX2NhbmNlbGxhdGlvbnNgLiBTbywgdGhlIG5ldyB2YXJpYWJsZSBzaG91bGQgYmUgYSAxIGlmIHRoZSBvcmlnaW5hbCBpcyBtb3JlIHRoYW4gMCBhbmQgMCBvdGhlcndpc2UuIE1ha2Ugc3VyZSB5b3UgZG8gdGhpcyBpbiBhIHdheSB0aGF0IGFjY291bnRzIGZvciB2YWx1ZXMgdGhhdCBtYXkgYmUgbGFyZ2VyIHRoYW4gYW55IHdlIHNlZSBpbiB0aGUgZGF0YXNldC4gIAoqIEZvciB0aGUgYGFnZW50YCBhbmQgYGNvbXBhbnlgIHZhcmlhYmxlcywgbWFrZSBuZXcgaW5kaWNhdG9yIHZhcmlhYmxlcyB0aGF0IGFyZSAxIGlmIHRoZXkgaGF2ZSBhIHZhbHVlIG9mIGBOVUxMYCBhbmQgMCBvdGhlcndpc2UuIEkgYWxzbyB1c2VkIGBzdGVwX211dGF0ZV9hdCgpYCBmb3IgdGhpcywgYnV0IHRoZXJlJ3MgbW9yZSB3YXlzIHlvdSBjb3VsZCBkbyBpdC4KKiBVc2UgYGZjdF9sdW1wX24oKWAgaW5zaWRlIGBzdGVwX211dGF0ZSgpYCB0byBsdW1wIHRvZ2V0aGVyIGNvdW50cmllcyB0aGF0IGFyZW4ndCBpbiB0aGUgdG9wIDUgbW9zdCBvY2N1cnJpbmcuIAoqIElmIHlvdSB1c2VkIG5ldyBuYW1lcyBmb3Igc29tZSBvZiB0aGUgbmV3IHZhcmlhYmxlcyB5b3UgY3JlYXRlZCwgdGhlbiByZW1vdmUgYW55IHZhcmlhYmxlcyB0aGF0IGFyZSBubyBsb25nZXIgbmVlZGVkLiAKKiBVc2UgYHN0ZXBfbm9ybWFsaXplKClgIHRvIGNlbnRlciBhbmQgc2NhbGUgYWxsIHRoZSBub24tY2F0ZWdvcmljYWwgcHJlZGljdG9yIHZhcmlhYmxlcy4gKERvIHRoaXMgQkVGT1JFIGNyZWF0aW5nIGR1bW15IHZhcmlhYmxlcy4gV2hlbiBJIHRyaWVkIHRvIGRvIGl0IGFmdGVyLCBJIHJhbiBpbnRvIGFuIGVycm9yIC0gSSdtIHN0aWxsIFtpbnZlc3RpZ2F0aW5nXShodHRwczovL2NvbW11bml0eS5yc3R1ZGlvLmNvbS90L3RpZHltb2RlbHMtc2VlLW5vdGVzLWVycm9yLWJ1dC1vbmx5LXdpdGgtc3RlcC14eHgtZnVuY3Rpb25zLWluLWEtY2VydGFpbi1vcmRlci8xMTUwMDYpIHdoeS4pCiogQ3JlYXRlIGR1bW15IHZhcmlhYmxlcyBmb3IgYWxsIGZhY3RvcnMvY2F0ZWdvcmljYWwgcHJlZGljdG9yIHZhcmlhYmxlcyAobWFrZSBzdXJlIHlvdSBoYXZlIGAtYWxsX291dGNvbWVzKClgIGluIHRoaXMgcGFydCEhKS4gIAoqIFVzZSB0aGUgYHByZXAoKWAgYW5kIGBqdWljZSgpYCBmdW5jdGlvbnMgdG8gYXBwbHkgdGhlIHN0ZXBzIHRvIHRoZSB0cmFpbmluZyBkYXRhIGp1c3QgdG8gY2hlY2sgdGhhdCBldmVyeXRoaW5nIHdlbnQgYXMgcGxhbm5lZC4KCmBgYHtyfQojIGhvdGVsX3JlY2lwZSA8LSByZWNpcGUoaXNfY2FuY2VsZWQgfiAuLAojICAgICAgICAgICAgICAgICAgICAgICAgZGF0YSA9IGhvdGVsX3RyYWluaW5nKSAlPiUKIyAgIHN0ZXBfbXV0YXRlX2F0KGNoaWxkcmVuLCBiYWJpZXMsIHByZXZpb3VzX2NhbmNlbGxhdGlvbnMsIGZuID0gfiBpZmVsc2UoLj4wLCAxLCAwKSkgJT4lCiMgICBzdGVwX25vcm1hbGl6ZShhbGxfbnVtZXJpY19wcmVkaWN0b3JzKCkpJT4lCiMgICBzdGVwX2R1bW15KGFsbF9ub21pbmFsKCksLWFsbF9vdXRjb21lcygpKQojIAojIGhvdGVsX3JlY2lwZSAlPiUKIyAgIHByZXAoaG90ZWxfdHJhaW5pbmcpICU+JQojICAganVpY2UoKQpgYGAKCmBgYHtyfQpob3RlbF9yZWNpcGU8LXJlY2lwZShpc19jYW5jZWxlZH4uLAogICAgICAgICAgICAgICAgICAgIGRhdGEgPSBob3RlbF90cmFpbmluZykgJT4lCiAgc3RlcF9tdXRhdGUoY2hpbGRyZW4gPSBpZmVsc2UoY2hpbGRyZW4gPiAwLCAxLCAwKSwKICAgICAgICAgICAgICAgICBiYWJpZXMgPSBpZmVsc2UoYmFiaWVzID4gMCwgMSwgMCksCiAgICAgICAgICAgICAgICAgcHJldmlvdXNfY2FuY2VsbGF0aW9ucyA9IGlmZWxzZShwcmV2aW91c19jYW5jZWxsYXRpb25zID4wLCAxLCAwKSwKICAgICAgICAgICAgICAgICBhZ2VudCA9IGlmZWxzZShhZ2VudCA9PSAiTlVMTCIsIDEsIDAgKSwKICAgICAgICAgICAgICAgICBjb21wYW55ID0gaWZlbHNlKGNvbXBhbnkgPT0gIk5VTEwiLCAxLCAwKSwKICAgICAgICAgICAgICAgICBjb3VudHJ5ID0gZmN0X2x1bXBfbihjb3VudHJ5LCBuPTUpICkgJT4lCiAgc3RlcF9ub3JtYWxpemUoYWxsX251bWVyaWNfcHJlZGljdG9ycygpKSU+JSNhbGxfcHJlZGljdG9ycygpLAogICAgICAgICAgICAgICAgICAgICAgIy1hbGxfbm9taW5hbCgpLAogICAgICAgICAgICAgICAgICAgICAgIy1hbGxfb3V0Y29tZXMoKSAlPiUKICBzdGVwX2R1bW15KGFsbF9ub21pbmFsKCksCiAgICAgICAgICAgICAgLWFsbF9vdXRjb21lcygpKSNhbGxfbm9taW5hbF9wcmVkaWN0b3JzCgpob3RlbF9yZWNpcGUgJT4lCiAgcHJlcChob3RlbF90cmFpbmluZykgJT4lCiAganVpY2UoKQpgYGAKCjUuIEluIHRoaXMgc3RlcCB3ZSB3aWxsIHNldCB1cCBhIExBU1NPIG1vZGVsIGFuZCB3b3JrZmxvdy4KCiogSW4gZ2VuZXJhbCwgd2h5IHdvdWxkIHdlIHdhbnQgdG8gdXNlIExBU1NPIGluc3RlYWQgb2YgcmVndWxhciBsb2dpc3RpYyByZWdyZXNzaW9uPyAoSElOVDogdGhpbmsgYWJvdXQgd2hhdCBoYXBwZW5zIHRvIHRoZSBjb2VmZmljaWVudHMpLiAgCgpVc2luZyByZWd1bGFyIGxvZ2lzdGljIHJlZ3Jlc3Npb24gbW9kZWxzIGlzIHBhcnRpY3VsYXJseSBwcm9ibGVtYXRpYyB3aGVuIHRoZXJlIGFyZSBhIGxhcmdlIG51bWJlciBvZiBmZWF0dXJlcywgd2hpY2ggY2FuIG1ha2UgdGhlIG1vZGVsIGxlc3MgaW50ZXJwcmV0YWJsZS4gVXNpbmcgTEFTU08sIHdlIGFyZSBhYmxlIHRvIGluY29ycG9yYXRlIGZlYXR1cmUgc2VsZWN0aW9uLCBzaW5jZSBpdCBjYW4gcHVzaCB0aGUgZWZmZWN0IG9mIGlycmVsZXZhbnQgZmVhdHVyZXMgdG93YXJkIHplcm8uIFRodXMsIGl0IGNhbiBub3Qgb25seSBpbXByb3ZlIHRoZSBtb2RlbCBidXQgaXQgYWxzbyBjb25kdWN0IGF1dG9tYXRlZCBmZWF0dXJlIHNlbGVjdGlvbi4KCiogRGVmaW5lIHRoZSBtb2RlbCB0eXBlLCBzZXQgdGhlIGVuZ2luZSwgc2V0IHRoZSBgcGVuYWx0eWAgYXJndW1lbnQgdG8gYHR1bmUoKWAgYXMgYSBwbGFjZWhvbGRlciwgYW5kIHNldCB0aGUgbW9kZS4gIAoKYGBge3J9CmhvdGVsX2xhc3NvX21vZCA8LSAKICBsb2dpc3RpY19yZWcobWl4dHVyZSA9IDEpICU+JSAKICBzZXRfZW5naW5lKCJnbG1uZXQiKSAlPiUgCiAgc2V0X2FyZ3MocGVuYWx0eSA9IHR1bmUoKSkgJT4lIAogIHNldF9tb2RlKCJjbGFzc2lmaWNhdGlvbiIpCmBgYAoKKiBDcmVhdGUgYSB3b3JrZmxvdyB3aXRoIHRoZSByZWNpcGUgYW5kIG1vZGVsLiAgCmBgYHtyfQpob3RlbF9sYXNzb193ZiA8LSAKICB3b3JrZmxvdygpICU+JSAKICBhZGRfcmVjaXBlKGhvdGVsX3JlY2lwZSkgJT4lIAogIGFkZF9tb2RlbChob3RlbF9sYXNzb19tb2QpCmBgYAoKNi4gSW4gdGhpcyBzdGVwLCB3ZSdsbCB0dW5lIHRoZSBtb2RlbCBhbmQgZml0IHRoZSBtb2RlbCB1c2luZyB0aGUgYmVzdCB0dW5pbmcgcGFyYW1ldGVyIHRvIHRoZSBlbnRpcmUgdHJhaW5pbmcgZGF0YXNldC4KCiogQ3JlYXRlIGEgNS1mb2xkIGNyb3NzLXZhbGlkYXRpb24gc2FtcGxlLiBXZSdsbCB1c2UgdGhpcyBsYXRlci4gSSBoYXZlIHNldCB0aGUgc2VlZCBmb3IgeW91LiAKKiBVc2UgdGhlIGBncmlkX3JlZ3VsYXIoKWAgZnVuY3Rpb24gdG8gY3JlYXRlIGEgZ3JpZCBvZiAxMCBwb3RlbnRpYWwgcGVuYWx0eSBwYXJhbWV0ZXJzICh3ZSdyZSBrZWVwaW5nIHRoaXMgc29ydCBvZiBzbWFsbCBiZWNhdXNlIHRoZSBkYXRhc2V0IGlzIHByZXR0eSBsYXJnZSkuIFVzZSB0aGF0IHdpdGggdGhlIDUtZm9sZCBjdiBkYXRhIHRvIHR1bmUgdGhlIG1vZGVsLgoqIFVzZSB0aGUgYHR1bmVfZ3JpZCgpYCBmdW5jdGlvbiB0byBmaXQgdGhlIG1vZGVscyB3aXRoIGRpZmZlcmVudCB0dW5pbmcgcGFyYW1ldGVycyB0byB0aGUgZGlmZmVyZW50IGNyb3NzLXZhbGlkYXRpb24gc2V0cy4gCiogVXNlIHRoZSBgY29sbGVjdF9tZXRyaWNzKClgIGZ1bmN0aW9uIHRvIGNvbGxlY3QgYWxsIHRoZSBtZXRyaWNzIGZyb20gdGhlIHByZXZpb3VzIHN0ZXAgYW5kIGNyZWF0ZSBhIHBsb3Qgd2l0aCB0aGUgYWNjdXJhY3kgb24gdGhlIHktYXhpcyBhbmQgdGhlIHBlbmFsdHkgdGVybSBvbiB0aGUgeC1heGlzLiBQdXQgdGhlIHgtYXhpcyBvbiB0aGUgbG9nIHNjYWxlLiAgCiogVXNlIHRoZSBgc2VsZWN0X2Jlc3QoKWAgZnVuY3Rpb24gdG8gZmluZCB0aGUgYmVzdCB0dW5pbmcgcGFyYW1ldGVyLCBmaXQgdGhlIG1vZGVsIHVzaW5nIHRoYXQgdHVuaW5nIHBhcmFtZXRlciB0byB0aGUgZW50aXJlIHRyYWluaW5nIHNldCAoSElOVDogYGZpbmFsaXplX3dvcmtmbG93KClgIGFuZCBgZml0KClgKSwgYW5kIGRpc3BsYXkgdGhlIG1vZGVsIHJlc3VsdHMgdXNpbmcgYHB1bGxfd29ya2Zsb3dfZml0KClgIGFuZCBgdGlkeSgpYC4gQXJlIHRoZXJlIHNvbWUgdmFyaWFibGVzIHdpdGggY29lZmZpY2llbnRzIG9mIDA/CgpgYGB7cn0Kc2V0LnNlZWQoNDk0KSAjIGZvciByZXByb2R1Y2liaWxpdHkKaG90ZWxfY3YgPC0gdmZvbGRfY3YoaG90ZWxfdHJhaW5pbmcsIHYgPSA1KQoKcGVuYWx0eV9ncmlkIDwtIGdyaWRfcmVndWxhcihwZW5hbHR5KCksCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgbGV2ZWxzID0gMTApCgpjdHJsX2dyaWQgPC0gY29udHJvbF9zdGFja19ncmlkKCkKI21ldHJpYyA8LSBtZXRyaWNfc2V0KHJtc2UpCgpob3RlbF9sYXNzb190dW5lIDwtIAogIGhvdGVsX2xhc3NvX3dmICU+JSAKICB0dW5lX2dyaWQoCiAgICByZXNhbXBsZXMgPSBob3RlbF9jdiwKICAgIGdyaWQgPSBwZW5hbHR5X2dyaWQsCiAgICAjbWV0cmljcyA9IG1ldHJpYywKICAgIGNvbnRyb2wgPSBjdHJsX2dyaWQKICAgICkKYGBgCgpgYGB7cn0KaG90ZWxfbGFzc29fdHVuZSAlPiUgCiAgY29sbGVjdF9tZXRyaWNzKCkgJT4lIAogIGZpbHRlcigubWV0cmljID09ICJhY2N1cmFjeSIpICU+JSAKICBnZ3Bsb3QoYWVzKHggPSBwZW5hbHR5LCB5ID0gbWVhbikpICsKICBnZW9tX3BvaW50KCkgKwogIGdlb21fbGluZSgpICsKICBzY2FsZV94X2xvZzEwKAogICBicmVha3MgPSBzY2FsZXM6OnRyYW5zX2JyZWFrcygibG9nMTAiLCBmdW5jdGlvbih4KSAxMF54KSwKICAgbGFiZWxzID0gc2NhbGVzOjp0cmFuc19mb3JtYXQoImxvZzEwIixzY2FsZXM6Om1hdGhfZm9ybWF0KDEwXi54KSkpICsKICBsYWJzKHggPSAicGVuYWx0eSIsIHkgPSAiYWNjdXJhY3kiKQpgYGAKIApgYGB7cn0KYmVzdF9wYXJhbSA8LSBob3RlbF9sYXNzb190dW5lICU+JSAKICBzZWxlY3RfYmVzdChtZXRyaWMgPSAiYWNjdXJhY3kiKQpiZXN0X3BhcmFtCmBgYAogCmBgYHtyfQpvbmVfc2VfcGFyYW0gPC0gaG90ZWxfbGFzc29fdHVuZSAlPiUgCiAgc2VsZWN0X2J5X29uZV9zdGRfZXJyKG1ldHJpYyA9ICJhY2N1cmFjeSIsIGRlc2MocGVuYWx0eSkpCgpob3RlbF9sYXNzb19maW5hbF93ZiA8LSBob3RlbF9sYXNzb193ZiAlPiUgCiAgZmluYWxpemVfd29ya2Zsb3cob25lX3NlX3BhcmFtKQpob3RlbF9sYXNzb19maW5hbF93ZgpgYGAKCmBgYHtyfQpob3RlbF9sYXNzb19maW5hbF9tb2QgPC0gaG90ZWxfbGFzc29fZmluYWxfd2YgJT4lIAogIGZpdChkYXRhID0gaG90ZWxfdHJhaW5pbmcpCgpob3RlbF9sYXNzb19maW5hbF9tb2QgJT4lIAogIHB1bGxfd29ya2Zsb3dfZml0KCkgJT4lIAogIHRpZHkoKSAKYGBgCgoKNy4gTm93IHRoYXQgd2UgaGF2ZSBhIG1vZGVsLCBsZXQncyBldmFsdWF0ZSBpdCBhIGJpdCBtb3JlLiBBbGwgd2UgaGF2ZSBsb29rZWQgYXQgc28gZmFyIGlzIHRoZSBjcm9zcy12YWxpZGF0ZWQgYWNjdXJhY3kgZnJvbSB0aGUgcHJldmlvdXMgc3RlcC4gCgoqIENyZWF0ZSBhIHZhcmlhYmxlIGltcG9ydGFuY2UgZ3JhcGguIFdoaWNoIHZhcmlhYmxlcyBzaG93IHVwIGFzIHRoZSBtb3N0IGltcG9ydGFudD8gQXJlIHlvdSBzdXJwcmlzZWQ/ICAKCmBgYHtyfQpob3RlbF9sYXNzb19maW5hbF9tb2QgJT4lIAogIHB1bGxfd29ya2Zsb3dfZml0KCkgJT4lIAogIHZpcCgpCmBgYAoKVGhlIG1vc3QgaW1wb3J0YW50IHZhcmlhYmxlIGlzIHJlc2VydmVkIHJvb20gdHlwZS4gWWVzIEkgYW0gc3VycHJpc2VkLgoKKiBVc2UgdGhlIGBsYXN0X2ZpdCgpYCBmdW5jdGlvbiB0byBmaXQgdGhlIGZpbmFsIG1vZGVsIGFuZCB0aGVuIGFwcGx5IGl0IHRvIHRoZSB0ZXN0aW5nIGRhdGEuIFJlcG9ydCB0aGUgbWV0cmljcyBmcm9tIHRoZSB0ZXN0aW5nIGRhdGEgdXNpbmcgdGhlIGBjb2xsZXRfbWV0cmljcygpYCBmdW5jdGlvbi4gSG93IGRvIHRoZXkgY29tcGFyZSB0byB0aGUgY3Jvc3MtdmFsaWRhdGVkIG1ldHJpY3M/CmBgYHtyfQojIEZpdCBtb2RlbCB3aXRoIGJlc3QgdHVuaW5nIHBhcmFtZXRlcihzKSB0byB0cmFpbmluZyBkYXRhIGFuZCBhcHBseSB0byB0ZXN0IGRhdGEKaG90ZWxfbGFzc29fdGVzdCA8LSBob3RlbF9sYXNzb19maW5hbF93ZiAlPiUgCiAgbGFzdF9maXQoaG90ZWxfc3BsaXQpCgojIE1ldHJpY3MgZm9yIG1vZGVsIGFwcGxpZWQgdG8gdGVzdCBkYXRhCmhvdGVsX2xhc3NvX3Rlc3QgJT4lIAogIGNvbGxlY3RfbWV0cmljcygpCmBgYAoKYGBge3J9CmNvbGxlY3RfbWV0cmljcyhob3RlbF9sYXNzb190dW5lKQpgYGAKClRoZSBhY2N1cmFjeSBvZiB0ZXN0aW5nIGRhdGEgc2xpZ2h0IGRlY3JlYXNlcyBjb21wYXJpbmcgdG8gY3Jvc3MtdmFsaWRhdGVkIG1ldHJpY3MuCgoqIFVzZSB0aGUgYGNvbGxlY3RfcHJlZGljdGlvbnMoKWAgZnVuY3Rpb24gdG8gZmluZCB0aGUgcHJlZGljdGVkIHByb2JhYmlsaXRpZXMgYW5kIGNsYXNzZXMgZm9yIHRoZSB0ZXN0IGRhdGEuIFNhdmUgdGhpcyB0byBhIG5ldyBkYXRhc2V0IGNhbGxlZCBgcHJlZHNgLiBUaGVuLCB1c2UgdGhlIGBjb25mX21hdCgpYCBmdW5jdGlvbiBmcm9tIGBkaWFsc2AgKHBhcnQgb2YgYHRpZHltb2RlbHNgKSB0byBjcmVhdGUgYSBjb25mdXNpb24gbWF0cml4IHNob3dpbmcgdGhlIHByZWRpY3RlZCBjbGFzc2VzIHZzLiB0aGUgdHJ1ZSBjbGFzc2VzLiBDb21wdXRlIHRoZSB0cnVlIHBvc2l0aXZlIHJhdGUgKHNlbnNpdGl2aXR5KSwgdHJ1ZSBuZWdhdGl2ZSByYXRlIChzcGVjaWZpY2l0eSksIGFuZCBhY2N1cmFjeS4gU2VlIHRoaXMgW1dpa2lwZWRpYV0oaHR0cHM6Ly9lbi53aWtpcGVkaWEub3JnL3dpa2kvQ29uZnVzaW9uX21hdHJpeCkgcmVmZXJlbmNlIGlmIHlvdSAobGlrZSBtZSkgdGVuZCB0byBmb3JnZXQgdGhlc2UgZGVmaW5pdGlvbnMuIEFsc28ga2VlcCBpbiBtaW5kIHRoYXQgYSAicG9zaXRpdmUiIGluIHRoaXMgY2FzZSBpcyBhIGNhbmNlbGxhdGlvbiAodGhvc2UgYXJlIHRoZSAxJ3MpLiAgCgpgYGB7cn0KcHJlZHMgPC0gY29sbGVjdF9wcmVkaWN0aW9ucyhob3RlbF9sYXNzb190ZXN0KSAKcHJlZHMgJT4lIAogIGNvbmZfbWF0KGlzX2NhbmNlbGVkLCAucHJlZF9jbGFzcykKYGBgCnRydWUgcG9zaXRpdmUgcmF0ZSAoc2Vuc2l0aXZpdHkp77yaCmBgYHtyfQoxNDI4Ni8oMTQyODYgKyAzMjk3KQpgYGAKdHJ1ZSBuZWdhdGl2ZSByYXRlIChzcGVjaWZpY2l0eSk6CmBgYHtyfQozNDMwNC8oMzQzMDQrNzgwNikKYGBgCiogVXNlIHRoZSBgcHJlZHNgIGRhdGFzZXQgeW91IGp1c3QgY3JlYXRlZCB0byBjcmVhdGUgYSBkZW5zaXR5IHBsb3Qgb2YgdGhlIHByZWRpY3RlZCBwcm9iYWJpbGl0aWVzIG9mIGNhbmNlbGluZyAodGhlIHZhcmlhYmxlIGlzIGNhbGxlZCBgLnByZWRfMWApLCBmaWxsaW5nIGJ5IGBpc19jYW5jZWxlZGAuIFVzZSBhbiBgYWxwaGEgPSAuNWAgYW5kIGBjb2xvciA9IE5BYCBpbiB0aGUgYGdlb21fZGVuc2l0eSgpYC4gCmBgYHtyfQpwcmVkcyAlPiUgCiAgZ2dwbG90KGFlcyh4PS5wcmVkXzEpKSArIAogIGdlb21fZGVuc2l0eShhZXMoZmlsbCA9IGlzX2NhbmNlbGVkKSwgYWxwaGEgPSAuNSwgY29sb3IgPSBOQSkKYGBgCgpBbnN3ZXIgdGhlc2UgcXVlc3Rpb25zOiAKYS4gV2hhdCB3b3VsZCB0aGlzIGdyYXBoIGxvb2sgbGlrZSBmb3IgYSBtb2RlbCB3aXRoIGFuIGFjY3VyYWN5IHRoYXQgd2FzIGNsb3NlIHRvIDE/ICAKClRoZSByZWQgcGFydCAoaXNfY2FuY2VsbGVkIDApIHdvdWxkIGJlIGFsbCBjbG9zZSB0byAwLCBtZWFuaW5nIC5wcmVkXzEgPSAwICwgd2hpbGUgdGhlIGJsdWUgcGFydCB3b3VsZCBiZSBjbG9zZSB0byAxLCBtZWFuaW5nIC5wcmVkXzEgPSAxLiAKCmIuIE91ciBwcmVkaWN0aW9ucyBhcmUgY2xhc3NpZmllZCBhcyBjYW5jZWxlZCBpZiB0aGVpciBwcmVkaWN0ZWQgcHJvYmFiaWxpdHkgb2YgY2FuY2VsaW5nIGlzIGdyZWF0ZXIgdGhhbiAuNS4gSWYgd2Ugd2FudGVkIHRvIGhhdmUgYSBoaWdoIHRydWUgcG9zaXRpdmUgcmF0ZSwgc2hvdWxkIHdlIG1ha2UgdGhlIGN1dG9mZiBmb3IgcHJlZGljdGVkIGFzIGNhbmNlbGVkIGhpZ2hlciBvciBsb3dlciB0aGFuIC41PyAKCkxvd2VyLiBTaW5jZSB0aGUgdHJ1ZSBwb3NpdGl2ZSByYXRlIGlzIHRoZSBwcm9iYWJpbGl0eSB0aGF0IGFuIGFjdHVhbCBwb3NpdGl2ZSB3aWxsIHRlc3QgcG9zaXRpdmUsIGlmIHRoZSBjdXRvZmYgaXMgbG93ZXIsIG1vcmUgY2FzZXMgd2lsbCBiZSBjbGFzc2ZpZWQgYXMgcG9zaXRpdmUgYW5kIHRodXMgdGhlIHRydWUgcG9zaXRpdmUgcmF0ZSB3aWxsIGJlIGhpZ2hlci4gIAoKYy4gV2hhdCBoYXBwZW5zIHRvIHRoZSB0cnVlIG5lZ2F0aXZlIHJhdGUgaWYgd2UgdHJ5IHRvIGdldCBhIGhpZ2hlciB0cnVlIHBvc2l0aXZlIHJhdGU/IAoKVGhlIHRydWUgbmVnYXRpdmUgcmF0ZSB3aWxsIGRlY3JlYXNlIHdoZW4gd2UgdHJ5IHRvIGdldCBhIGhpZ2hlciB0cnVlIHBvc2l0aXZlIHJhdGUuIFRoZXkgYXJlIG5lZ2F0aXZlbHkgY29ycmVsYXRlZC4gCgo4LiBMZXQncyBzYXkgdGhhdCB0aGlzIG1vZGVsIGlzIGdvaW5nIHRvIGJlIGFwcGxpZWQgdG8gYm9va2luZ3MgMTQgZGF5cyBpbiBhZHZhbmNlIG9mIHRoZWlyIGFycml2YWwgYXQgZWFjaCBob3RlbCwgYW5kIHNvbWVvbmUgd2hvIHdvcmtzIGZvciB0aGUgaG90ZWwgd2lsbCBtYWtlIGEgcGhvbmUgY2FsbCB0byB0aGUgcGVyc29uIHdobyBtYWRlIHRoZSBib29raW5nLiBEdXJpbmcgdGhpcyBwaG9uZSBjYWxsLCB0aGV5IHdpbGwgdHJ5IHRvIGFzc3VyZSB0aGF0IHRoZSBwZXJzb24gd2lsbCBiZSBrZWVwaW5nIHRoZWlyIHJlc2VydmF0aW9uIG9yIHRoYXQgdGhleSB3aWxsIGJlIGNhbmNlbGluZyBpbiB3aGljaCBjYXNlIHRoZXkgY2FuIGRvIHRoYXQgbm93IGFuZCBzdGlsbCBoYXZlIHRpbWUgdG8gZmlsbCB0aGUgcm9vbS4gSG93IHNob3VsZCB0aGUgaG90ZWwgZ28gYWJvdXQgZGVjaWRpbmcgd2hvIHRvIGNhbGw/IEhvdyBjb3VsZCB0aGV5IG1lYXN1cmUgd2hldGhlciBpdCB3YXMgd29ydGggdGhlIGVmZm9ydCB0byBkbyB0aGUgY2FsbGluZz8gQ2FuIHlvdSB0aGluayBvZiBhbm90aGVyIHdheSB0aGV5IG1pZ2h0IHVzZSB0aGUgbW9kZWw/IAoKVGhlIGhvdGVsIGNhbiB1c2UgdGhlIG1vZGVsIHJlc3VsdCB0byBsb29rIGF0IHdobyB0byBjYWxsLCBpZiB0aGUgbW9kZWwgcHJlZGljdHMgdGhlIGN1c3RvbWVyIGFzIGNhbmNlbGxlZCwgdGhlbiBpdCBwcm9iYWJseSBzaG91bGQgbWFrZSBhIHBob25lIGNhbGwgdG8gdGhpcyBwZXJzb24uIAoKV2UgY291bGQgbG9vayBhdCB0aGUgcmVzdWx0cyBmcm9tIApjb2xsZWN0X3ByZWRpY3Rpb25zIGFuZCBjb21wYXJlIHRoZSBudW1iZXJzIG9mIC5wcmVkXzAgYW5kIC5wcmVkXzEuIElmIHdlIGFyZSBnZXR0aW5nIGEgMSBpbiAucHJlZF9jbGFzcywgd2Ugc2hvdWxkIGxvb2sgYXQgLnByZWRfMSB0byBzZWUgdGhlIGNoYW5jZSBvZiBnZXR0aW5nIDEsIGlmIGl0IGlzIGhpZ2gsIHRoZW4gd2Uga25vdyB0aGF0IHRoaXMgcGVyc29uIGlzIHdvcnRoIHRoZSBlZmZvcnQgdG8gZG8gdGhlIGNhbGxpbmcgc2luY2UgdGhlIGNoYW5jZSBvZiBnZXR0aW5nIGEgMSBpcyBoaWdoLiAKIApBbm90aGVyIHdheSB0byB1c2UgdGhpcyBtb2RlbCBpcyB0byBsb29rIGF0IGZlYXR1cmUgaW1wb3J0YW5jZSwgYW5kIHdpdGhvdXQgYWNjdWFsbHkgaW1wbGVtZW50aW5nIHRoZSBtb2RlbCBvbiBuZXcgZGF0YSwgd2UgbWlnaHQgYmUgYWJsZSB0byBpbmZlciBzb21ldGhpbmcgZnJvbSBpbXBvcnRhbnQgZmVhdHVyZXMuIAoKOS4gSG93IG1pZ2h0IHlvdSBnbyBhYm91dCBxdWVzdGlvbmluZyBhbmQgZXZhbHVhdGluZyB0aGUgbW9kZWwgaW4gdGVybXMgb2YgZmFpcm5lc3M/IEFyZSB0aGVyZSBhbnkgcXVlc3Rpb25zIHlvdSB3b3VsZCBsaWtlIHRvIGFzayBvZiB0aGUgcGVvcGxlIHdobyBjb2xsZWN0ZWQgdGhlIGRhdGE/IAoKSXMgdGhlIGRpc3RyaWJ1dGlvbiBvZiByYWNlLCBhZ2UgYW5kIGdlbmRlciBldmVuIGluIHRoZSBkYXRhc2V0PyBJcyB0aGVyZSBhbnkgZGlzY3JpbWluYXRpb24gb2YgYWdlLCBhZ2UgYW5kIGdlbmRlciBpbiB0aGUgZGF0YSBjb2xsZWN0aW9uIHByb2Nlc3M/IElzIHRoZSBkYXRhIGNvbGxlY3RlZCBldGhpY2FsbHkgYW5kIHByb3Blcmx5IHdpdGggY29uc2VudCBmcm9tIGNvbmNlcm5pbmcgcGFydGllcz8gCgojIyBCaWFzIGFuZCBGYWlybmVzcwoKUmVhZCBbQ2hhcHRlciAxOiBUaGUgUG93ZXIgQ2hhcHRlcl0oaHR0cHM6Ly9kYXRhLWZlbWluaXNtLm1pdHByZXNzLm1pdC5lZHUvcHViL3ZpOG9ieGg3L3JlbGVhc2UvNCkgb2YgRGF0YSBGZW1pbmlzbSBieSBDYXRoZXJpbmUgRCdJZ25hemlvIGFuZCBMYXVyZW4gS2xlaW4uIFdyaXRlIGEgNC02IHNlbnRlbmNlIHBhcmFncmFwaCByZWZsZWN0aW5nIG9uIHRoaXMgY2hhcHRlci4gQXMgeW91IHJlZmxlY3QsIHlvdSBtaWdodCBjb25zaWRlciByZXNwb25kaW5nIHRvIHRoZXNlIHNwZWNpZmljIHF1ZXN0aW9ucy4gV2Ugd2lsbCBhbHNvIGhhdmUgYSBkaXNjdXNzaW9uIGFib3V0IHRoZXNlIHF1ZXN0aW9ucyBpbiBjbGFzcyBvbiBUaHVyc2RheS4KClRoZSBsYXJnZXIgaXNzdWUgb2YgcG93ZXIgYW5kIHByaXZpbGVnZSBpcyBub3Qgb25seSBldmVyeXdoZXJlIGluIG91ciBsaWZlLiBJdCBpcyBhbHNvIGluIHRoZSBkYXRhIGNvbGxlY3Rpb24gYW5kIHByb2Nlc3NpbmcgcHJvY2Vzc2VzIGluIHRoZSBhY2FkZW1pYyBjb21tdW5pdHkgd2hpY2ggbGVhZHMgdGhlIHByb2JsZW1zIGFuZCBtaW5vcml0aWVzIHVuYWRkcmVzc2VkLiBXZSBuZWVkIHRvIGJlIGF3YXJlIG9mIHRoZSBmb3VyIGRvbWFpbnMgb2YgbWF0cml4IG9mIGRvbWluYXRpb24gd2hlbiBjb2xsZWN0aW5nIGFuZCBwcm9jZXNzaW5nIHRoZSBkYXRhLiBTaW5jZSB0aGUgZGVtb2dyYXBoaWNzIG9mIGRhdGEgc2NpZW5jZSBkb2VzIG5vdCByZXByZXNlbnQgdGhlIHBvcHVsYXRpb24sIGFuZCB0aG9zZSB3aG8gd29ya3MgaW4gdGhlIGluZHVzdHJ5IGRvIG5vdCByZWZsZWN0IHRoZSBwb3B1bGF0aW9uIGFzIHdlbGwsIGl0IGlzIG5lY2Vzc2FyeSB0byBrZWVwIGluIG1pbmQgdG8gYWR2b2NhdGUgZm9yIHRoZSB2b2ljZXMgb2YgdGhvc2UgdW5kZXJyZXByZXNlbnQuIFdlIG5lZWQgdG8gZGVzaWduIG1vZGVscyBhbmQgYWxnb3JpdGhtcyB0aGF0IGhlbHAgcGVvcGxlIGRvIGFsbGV2aWF0ZSB0aGUgcHJvYmxlbXMgb2YgZ2VuZGVyIGFuZCByYWNpYWwgYmlhcy4gCgoqIEF0IHRoZSBlbmQgb2YgdGhlICJNYXRyaXggb2YgRG9taW5hdGlvbiIgc2VjdGlvbiwgdGhleSBlbmNvdXJhZ2UgdXMgdG8gImFzayB1bmNvbWZvcnRhYmxlIHF1ZXN0aW9uczogd2hvIGlzIGRvaW5nIHRoZSB3b3JrIG9mIGRhdGEgc2NpZW5jZSAoYW5kIHdobyBpcyBub3QpPyBXaG9zZSBnb2FscyBhcmUgcHJpb3JpdGl6ZWQgaW4gZGF0YSBzY2llbmNlIChhbmQgd2hvc2UgYXJlIG5vdCk/IEFuZCB3aG8gYmVuZWZpdHMgZnJvbSBkYXRhIHNjaWVuY2UgKGFuZCB3aG8gaXMgZWl0aGVyIG92ZXJsb29rZWQgb3IgYWN0aXZlbHkgaGFybWVkKT8iIEluIGdlbmVyYWwsIGhvdyB3b3VsZCB5b3UgYW5zd2VyIHRoZXNlIHF1ZXN0aW9ucz8gQW5kIHdoeSBhcmUgdGhleSBpbXBvcnRhbnQ/ICAKCkkgd291bGQgdGhpbmsgYWJvdXQgd2hvIGFyZSB0aGUgbWlub3JpdGllcyBpbiB0aGUgaW5kdXN0cnkgdGhhdCBpcyB1bmRlcnJlcHJlc2VudCwgYW5kIGFsc28gdGhpbmtzIGFib3V0IHRoZSBtYWpvcml0eSBpbiB0aGUgaW5kdXN0cnkgYW5kIHRoZWlyIGNvbmZsaWN0aW5nIGludGVyZXN0cyB0aGF0IG1pZ2h0IGxlYWQgdG8gZGlmZmVyZW50IHJlc3VsdHMgd2hlbiBhbmFseXppbmcgZGF0YS4gVGhlIG1ham9yaXRpZXMnIHBvaW50IG9mIHZpZXcgbWlnaHQgb2Z0ZW4gYmUgcHJpb3JpdGl6ZWQgYmVjYXVzZSB0aGV5IGhhdmUgbW9yZSBwb3dlciwgd2hpbGUgdGhlIG1pbm9ydGllcyBhcmUgbm90LiBUaGVzZSBxdWVzdGlvbnMgYXJlIGltcG9ydGFudCBmb3IgdGhlIGZhaXJuZXNzIG9mIHdoYXQgd2UgYXJlIGRvaW5nLiAKCiogQ2FuIHlvdSB0aGluayBvZiBhbnkgZXhhbXBsZXMgb2YgbWlzc2luZyBkYXRhc2V0cywgbGlrZSB0aG9zZSBkZXNjcmliZWQgaW4gdGhlICJEYXRhIFNjaWVuY2UgZm9yIFdob20/IiBzZWN0aW9uPyBPciB3YXMgdGhlcmUgYW4gZXhhbXBsZSB0aGVyZSB0aGF0IHN1cnByaXNlZCB5b3U/ICAKQXNpYW5zIHRoYXQgYmVpbmcgZGlzY3JpbWludGF0ZWQgZnJvbSBjb2xsZWdlIGFkbWlzc2lvbiBiZWNhdXNlIHRoZXkgYXJlIGFzaWFucy4gCgoqIEhvdyBkaWQgdGhlIGV4YW1wbGVzIGluIHRoZSAiRGF0YSBTY2llbmNlIHdpdGggV2hvc2UgSW50ZXJlc3RzIGFuZCBHb2Fscz8iIHNlY3Rpb24gbWFrZSB5b3UgZmVlbD8gV2hhdCByZXNwb25zaWJpbGl0eSBkbyBjb21wYW5pZXMgaGF2ZSB0byBwcmV2ZW50IHRoZXNlIHRoaW5ncyBmcm9tIG9jY3VycmluZz8gV2hvIGlzIHRvIGJsYW1lPwoKSXQgbWFrZXMgbWUgZmVlbCB0aGF0IGFza2luZyB0aGVzZSBxdWVzdGlvbnMgYW5kIGNvbnN0YW50bHkgdGhpbmtpbmcgb2YgdGhlc2UgcmVzdWx0cyBhcmUgaW1wb3J0YW50LiBUaGUgY29tcGFuaWVzIGhhdmUgdGhlIHJlc3BvbnNpYmlsaXR5IGJ5IGFkZHJlc3Npbmcgd2hvIGlzIHRoZSBwcml2aWxlZ2VkIHBhcnR5IGFuZCB3aG8gaXMgYmVpbmcgb3Zlcmxvb2tlZC4gVGhlIHdob2xlIHNvY2lldHkgYW5kIGluZHVzdHJ5IGlzIHRvIGJsYW1lLCB0byBiZSBob25lc3QsIGFuZCB0aGUgY2hhbmdlIG5lZWRzIHRvIGNvbWUgZnJvbSB0aGUgZWR1Y2F0aW9uIHN5c3RlbS4gCgo=